Method for determining the direction in which a user is looking

ABSTRACT

The present invention relates to a method for determining the direction in which a user is looking, which includes acquiring images of the eye, in particular by means of an optical sensor, the method including: a) a first processing of the images, yielding information on the orientation of the eye according to the observation of an area of the eye in which the aspect varies with the rotation of the eye; and b) a second processing of the images, yielding information on the kinematics of the eye by comparing at least two consecutive images; method in which information is generated relating to the direction in which the user is looking relative to the head of the user, at least according to the information supplied by the first and second processes.

The present invention relates to the determination of the direction of aperson's gaze by tracking their ocular movements, in particular whenthis person is viewing a screen of a computing system.

U.S. Pat. No. 6,113,237 pertains to a device making it possible todetect the horizontal and vertical movements of a person's eyes. Itgives the position of the pupil, but not the direction of the gaze. Itdoes not make it possible to obtain high precision.

Patent applications US 2005/0110950 and US 2010/0283972 pertain tomedical applications of the detection of the saccadic movements of theeye and do not teach more particularly to determine the direction of thegaze.

In international application WO 2010/083853, an image of the sceneobserved by the user is acquired so as to determine the latter'sobservation point.

Application US 2010/045933 proposes a scheme for identifying thedirection of the gaze involving a reflection on the retina whichrequires a complex optical rig (mobile mirrors scanning the zonemeasured on the basis of a pointlike luminous source of very small size)associated with a scheme for off-line processing of the informationbased on fixed frames of reference. This device does not use thesynergistic and auto-sustained aspects of a real-time dual source ofinformation with relative referencing and may not therefore respond in asatisfactory manner to the problematic issues of instantaneousprocessing of information and of auto-correction of drift.

Application EP 2 261 772 describes a scheme for reconstructing thedirection of gaze on the basis of a stationary, remote camera facing theuser. A first processing identifies and then tracks by optical flow theposition and the orientation of the head and a second processingidentifies one or both of the user's irises. This fixed device, which isnot embedded on board, appreciably restricts the user's mobility anddoes not meet the expected precision constraints of such a system.

Application US 2012/019645 describes a device making it possible todetermine the gaze data on an onboard viewing screen. It does notinclude any referencing to the outside world.

Existing systems may require relatively lengthy and constrainingcalibration. Moreover, drift is liable to occur, which limits in theduration the precision obtained and therefore the possibility of makingenduring use of the results obtained.

A need exists to improve the productivity and the precision of theinterface between man and a computing system, and in particular tosimplify the calibration process.

A need further exists to accelerate the processing of the data and torender the system for determining the direction of the gaze morereactive and easily usable in real time.

Finally, a need exists to have a method allowing the user to benefitfrom a certain freedom of movement, at least of the head.

The invention is aimed at meeting all or part of these needs.

Method

The subject of the invention is thus, according to a first of itsaspects, a method for determining the direction of the gaze of a usercomprising the acquisition of images of the eye, in particular with theaid of an optical sensor, preferably a camera, the method comprising:

a) a first processing of these images which provides an item ofinformation about the orientation of the eye, on the basis of theobservation of a zone of the eye whose aspect varies with the rotationof the eye, and

b) a second processing of these images which provides an item ofinformation about the kinematics of the eye by comparing at least twosuccessive images,

in which method an item of information relating to the direction of thegaze in relation to the user's head is generated, in particular in theframe of reference of the user's head on the basis at least of the itemsof information delivered by the first and second processings.

By virtue of the aggregate of these processings, the precision isimproved; this makes it possible to decrease the resolution of thecamera used to acquire the images and therefore to increase theprocessing speed and to simplify the processing; this makes it possibleto reduce the electrical consumption or to lighten the system or torender the system wireless and to increase its energy self-sufficiency.

Furthermore, the better precision makes it possible to reduce the driftover time with respect to the initial calibration and thus to prolongthe duration of use without needing a new calibration. Comfort of use isthereby enhanced.

The first and second processings do not necessarily take place in theorder a) then b) and may also take place in the order b) then a), theprocessings being carried out according to a loop making it possible toprovide at regular time intervals an item of information relating to thedirection of the gaze.

The image acquisition scheme used in the method may advantageously benon-intrusive of the eye. Stated otherwise, the images serve todetermine the position and/or the displacement of the eye are images ofthe exterior of the eye and not images of a deep surface of the eye,such as the retina or the macula, or involving refractions and/orreflections on elements of the eye, such as the retina or the cornea.Thus, the images usable in the invention are eye surface aspect images,such as a portion at least of the contour of the iris and/or of thesclera. This may make it possible to preserve a relatively simpleacquisition optic which is easy to integrate into spectacles forexample. This may also make it possible to simplify the lightingintegrated into the device worn by the user, since it is not necessaryto cause a light ray to penetrate within the eye under particularconditions of incidence.

Diffuse lighting of the surface of the eye, in particular of the irisand of the sclera, may be suitable.

The invention does not rely on the observation of the reflection ofluminous sources on the eye and may thus be more independent of theluminous environment. The images acquired may represent externalkinematic and/or anatomical data of the eye.

The first and second processings may rely on the acquisition of imagesat different acquisition frequencies, and preferably the firstprocessing is performed at a first acquisition frequency smaller thanthe second processing which is performed at a second frequency, withpreferably furthermore a factor of at least 10 between the first andsecond frequencies.

The first and second processings may rely on the acquisition of imagesat different resolutions, and preferably the first processing relies onthe acquisition of images at a greater resolution than the resolution ofthe acquisition performed during the second processing, for examplegreater by a factor of 5 at least.

The item of information delivered by the first processing may make itpossible to correct the drifts related to the second processing, withinthe precision of the measurement.

First Processing

The first processing provides an item of information about theorientation of the eye, on the basis of the observation of a zone of theeye whose aspect varies with the rotation of the eye. The orientation isdetermined in a static manner independently of its evolution over time.

This first processing may be performed on the basis of a single image.It may be performed for each image.

This first processing may comprise the determination of the parametersrelated to the shape of the ellipse corresponding to the pupil, inparticular so as to determine the position of a point of the eye, forexample of the center of the pupil. It is possible to use an algorithmfor reconstructing the shape of the pupil in the image by determining,by processing the image, the parameters of the ellipse corresponding tothe circle of the pupil observed in a direction making an angle with thenormal to the plane of this circle. It is possible to deduce theorientation of the eye by analyzing the evolution of the shape of thisellipse.

Such an algorithm is known from the article “Matching 2-D ellipses to3-D circles with application to vehicle pose identification” by M.Hutter et al. Image and Vision Computing New Zealand, 2009. IVCN '09.

The reconstruction of an ellipse is for example possible on the basis offive points of its contour.

The disk of the pupil, projected onto a plane, is an ellipse which maybe parametrized in the general form: a.x²+b.x.y+c.y²+d.x+e.y+f=0, withb²−4.a.c<0. A limited number of points of the contour, in particularfive, suffice to calculate the descriptive parameters of this ellipse(center, minor axis, major axis and angle of rotation). Knowing thesecharacteristics and those of the camera, it is thus possible toreconstruct in 3D the direction of the vector normal to the disk whoseprojection is the ellipse.

The reconstruction of the ellipse may require a minimum resolution inorder to obtain the necessary precision. In the case where theresolution of the image is low, use is made of an algorithm forreconstructing the ellipse making it possible to take account in thewhole of the image of the intensity of the gray levels, rather thanbeing concentrated on the contour of the spot corresponding to the pupilto retrace the ellipse there. One thereafter seeks to match a continuousfunction of given form with the corresponding gray levels. A betterresolution than the resolution of the image is thus obtained dependingon the size and the number of pixels therein. It is thus possible to usea relatively low resolution while retaining sufficient precision.

The direction of the eye in the frame of reference of the camera isgiven by the normal to the disk of the pupil taken at its center. Thisnormal and also the center of the disk, which is different from thecenter of the ellipse, are obtained on the basis of the characteristicsof the ellipse. The direction of the gaze in the frame of reference ofthe head is obtained by applying a fixed and known 3-axis rotationmatrix.

The processed image does not necessarily comprise a view of the iris ofthe eye in its entirety. The invention makes it possible to obtain theellipse even when the image comprises only a portion of the iris of theeye, either because the eye was partially closed during the capture ofthe image, or because the image is captured on the eye with asignificant magnification. The processing may make it possible to obtainthe desired item of information, even if only a partial view of theboundary of the iris, or indeed only of a boundary of the latter withthe white of the eye, is available.

In contradistinction to the conventional schemes which calculate theorientation of the eye as a nonlinear function of the distance betweenthe center of the ellipse and the center of one of the reflections onthe cornea of an external lighting, called Purkinje points), this schememakes it possible to ascertain the orientation of the eye withoutcalling upon an external frame of reference such as the Purkinje points.The linearity of this solution makes it possible to dispense with thecalibration schemes inherent in the conventional scheme.

Second Processing

The second processing provides an item of information about thekinematics of the eye by comparing at least two successive images, thatis to say an item of information relating to the evolution of theposition of the eye in its orbit between two instants.

This second processing may be performed on the basis of two consecutiveimages. It may be performed for each pair of successive images acquiredby the optical sensor.

The second processing makes it possible to obtain items of informationabout the displacement of the eye in its orbit, and in particular anitem of information about the angle of rotation of the eye, by makingthe assumption that the eye is a sphere.

The second processing comprises the determination of the optical flowbetween two successive images of the eye, that is to say the apparentmovement caused by the relative movement between the optical sensorwhich acquires images and the eye. A corresponding algorithm is knownfrom the article “Determining Optical Flow” by Berthold K. P. Horn etal, Artificial Intelligence, vol. 17, pp. 185-203, 1981.

The variation of the flow of pixels in an image is measured, bydifferencing between two or more images. Two successive images may bevery close together in time. Two successive images are preferably ofconstant intensity. It is possible to chart between two images of oneand the same zone of a mobile part of the eye comprising contrasts, forexample two successive images of the sclera comprising blood vessels, orany region including all or part of the iris or of the pupil, the choiceof the region not being limiting, the variation of the location ofcharacteristic zones through their shapes or intensity.

The variation of location of the characteristic zones is observedthrough the variation of the flow of intensity of pixels in the planeparallel to the plane of the sensor. Measurement of this variation ofthe optical flow of pixels does not make it necessary to have toexplicitly identify a reference image or reference patterns orcharacteristics in the image. In particular these “images” may becaptured at very high frequencies by a second sensor independent of thefirst processing. It is not necessary for this second sensor to exhibitgood resolution or good focusing provided that the measured zone becontrasted. Measurement of this variation of the optical flow isfacilitated if the intensity of each image is constant.

Combination

The items of information resulting from the two processings may becombined to generate the item of information relating to the directionof the gaze, which may be an item of information relating to thedirection of an eye. The item of information relating to the directionof the gaze may result from the addition of the two items ofinformation, each being weighted as a function of a law which may dependon the two items of information themselves. The first processing makesit possible to obtain an item of information relating to a change oforientation of the direction of the gaze, in the frame of reference ofthe camera, through the measurement of the normal to the pupil. Thesecond processing makes it possible to ascertain the differential inmovement of one or more points of the sphere modeling the eye in theplane of the camera. These two items of information stem from twoindependent processings. They are combined and weighted to obtain afinal eye direction measurement. This weighting is dependent on theconsistency of the movement at this instant. Under the assumption of apure three-dimensional rotational movement in relation to two axes andof unknown center and radius, from an initial angular positioncorresponding to the image at an instant t, the solution for therotation making it possible to obtain the final state at the instantt+dt, which may be the following image, according to the two items ofinformation obtained, is optimized. The solution retained is a weightingof the two results as a function of their consistency with the rotationmodel imposed. The two processings may be performed simultaneously orquasi-simultaneously. This may make it possible to profit from theirfunctional and/or topographical complementarity so as to accelerate orfacilitate the processings or improve their quality. One of the items ofinformation may make it possible to correct the other or to help todetermine it. It is for example possible to proceed by interpolation todetermine one of the items of information.

It is for example possible by virtue of these items of information tocircumvent the corneal refraction which may produce non-linearitiesstemming from the refraction of the pupil on the cornea, this refractionbeing due to the position of the optical sensor. The second processingmakes it possible to obtain the items of information about thedisplacement of the eye in its orbit with a smaller influence of therefraction, insofar as the eye zone used is preferably situatedrelatively in the corner of the eye, for example on the sclera.

The second processing may produce errors in the case where the eyerotates too quickly in relation to the speed of capture of the opticalsensors, hence the benefit of using temporally close successive images,therefore an optical sensor with high acquisition frequency, and ofcombining with the first processing.

It is for example possible to perform a modeling of the location of thecenter of rotation by considering that all of the already determineddirections of the gaze pass through the center of the eye. Thus, it ispossible to facilitate the determination of the subsequent directions ofthe gaze, by presupposing that they must also pass through the modeledcenter of rotation. Through the first processing, the position of thecenter of the pupil and the normal to the latter are known at eachinstant. On the basis of several successive measurements, the center ofrotation of the pupil in space is the solution which minimizes, forexample by the least squares scheme, the equation of a sphere knowingthe positions in the plane of the camera of several points of thissphere as well as the normal to the sphere at each of these points.

The synergy of these two processings makes it possible to optimize theadvantages of each in their inherent regimes.

The first processing makes it possible to precisely ascertain theposition of the pupil when the movement of the eye is slow or zerocompared with the acquisition frequency, angular speed corresponding toa movement frequency of about 30 Hz for example. If the displacement isvery fast, in particular with an angular speed corresponding to afrequency of greater than or equal to 200 Hz, the image is distorted,appears fuzzy, scrambled and the reconstruction bad.

The second processing measures a rotation speed. The angular orientationis obtained by integration of this speed over time. The acquisitionsystem being much faster than the movement of the eye, this schemeallows precise measurement during the very fast movements of the eye.Conversely, it is less beneficial during slow movements of the eye sincethe background noise of the image becomes significant in regard to thesmall displacement.

The combining of the two processings is therefore particularlyadvantageous and makes it possible to obtain at one and the same timegood spatial precision and good temporal precision of the orientation ofthe eye by allowing a reciprocal autocorrection of the two processings.

Device

It is advantageously possible to use for the acquisition of the images adevice worn by the user, comprising at least one first camera configuredto acquire an image of all or part of an eye of the user.

The device may also comprise an electronic circuit making it possible toprocess the images acquired according to the method according to theinvention, so as to determine a relative displacement of the eye withrespect to the camera and the evolution of the direction of gaze overtime.

This device may be worn in all or part by the user.

The device may furthermore comprise at least one informationrepresentation system in particular of the type such as a screen orsemi-transparent screen, projector, loudspeaker or earpiece, forcefeedback vibratory system. The representation system may thus representfor example in a visual, graphical, audible or other manner an item ofinformation obtained in step a) or b) or deriving from such an item ofinformation.

The device may furthermore comprise at least one sensor of an item ofphysiological information relating to the wearer of the device. Thelatter sensor is for example chosen from among a microphone, a motion oracceleration sensor, an image sensor, one or more electrodes, aperspiration sensor.

The processing may take place in a manner entirely integrated intospectacles worn by the user, for example, or in an offloaded manner, byvirtue of remote transmission of the images for example, or else in amixed manner, in part at the level of the spectacles and in part in anoffloaded manner, a preprocessing being for example performed by virtueof an electrical circuit carried by the spectacles and an item ofinformation dispatched to a remote circuit through a non-wire link. Thepreprocessing makes it possible to reduce the bitrate of datatransferred.

For the acquisition of the images, the eye or both eyes preferably is orare illuminated with infrared lighting, in particular with one or moreLEDs. The luminous source or sources are for example of relatively wideangular aperture, in particular greater than 60°, so as to cover a largezone of the eye. The lighting source or sources may be disposed on thesides, in particular the exterior sides, this advantageously makes itpossible to minimize the reflections observed by the camera.

The device may comprise just a single optical sensor to acquire an imageof the user's eye. In the case where the device comprises just a singleoptical sensor and is devoid of a second optical sensor, the first andonly optical sensor may by itself make it possible to obtain the imagesnecessary for the two processings.

The successive images taken by the optical sensor or sensors may beimages of the corner of the eye and/or of the pupil and/or of thesclera.

The device may comprise at least one second optical sensor configured toacquire an image of all or part of the eye.

The first and second optical sensors may allow the acquisition of imagesof one and the same eye of the user. The use of two sensors for an eyeallows less dependence on possible significant variations of thedirection of gaze.

The first and second optical sensors are each directed toward adifferent part of one and the same eye of the user, for example thepupil and the sclera.

The acquisition and processing chain associated with one eye may beduplicated for the other eye. Thus, the device may comprise at least onethird optical sensor worn by the user and configured to acquire an imageof all or part of the user's second eye. The device may comprise afourth optical sensor worn by the user and configured to acquire animage of all or part of user's second eye. The third and the fourthoptical sensor are each directed toward a different part of the user'ssecond eye, for example the pupil and the sclera. Thus, it is possibleto undertake simultaneous observation of the two eyes, thereby making itpossible to further improve the precision.

It is furthermore possible to detect the frequency of closing of theeyelid of the eye studied or of the eyes studied. It is possible to usean electromyogram and/or an electroocculogram. The closing of one orboth eyelids may be used to indicate that the user validates thepositioning of the mouse at the targeted point of the screen, statedotherwise that he clicks with his eyes on the screen. The user may forexample have to close his eyes fairly tightly to indicate that hedesires to validate a choice of position.

Sensors

The sensor or sensors, in particular the second and/or fourth sensors,may be sufficiently fast digital sensors, for example operating at afrequency of at least 200 Hz, or indeed of at least 500 Hz, or indeed ofat least 1000 Hz, for example at a frequency of about 1500 Hz, or indeeda frequency of at least 2000 Hz.

The sensor or sensors, in particular the second and/or fourth sensors,may have a relatively low resolution, for example of less than 500*500pixels, or indeed of less than 200*200 pixels, for example lying between16*16 and 100*100 pixels.

A captured image may for example have a resolution of the order of100*100 pixels, or indeed of 32*32 pixels, or else indeed of 24*24pixels.

Such a resolution may be sufficient to make it possible to obtainsatisfactory images, while making it possible not to slow down thedetermination of the calculation of the direction of the user's gaze.

The sensor or sensors may be thermal or based on infrared radiation. Thesensor or sensors may be associated with one or more LEDs. The sensor orsensors may comprise a camera, in particular RGB. The camera or camerasmay be thermal.

The camera or cameras may be configured so that the eye is situated inthe focal plane of the camera.

The sensor or sensors are preferably disposed on the device otherwisethan centered with respect to an eye. They are preferably distant from aplane passing through the pupil of the corresponding eye.

The placement of the sensors may make it possible to obtain sufficientcontrast between two acquired images so as to view a difference betweenthe two images. The sensors are preferably disposed sufficiently farfrom the eyelashes. The sensor or sensors may make it possible tocapture a zone of the eye of a size of about 5 mm by 5 mm. The sensor orsensors may for example be disposed at a distance from the eye ofbetween 1 cm and 5 cm, for example of the order of 2 cm. The choice ofthe distance makes it possible to ensure that the sensor is sufficientlyclose to the eye, and therefore sees only the eye, so as to improve theprecision.

The sensor or sensors may be configured to preferably provide weaklylocalized lighting, stated otherwise rather diffuse lighting. Indeed, itis sought to avoid a reflection of the luminous source from beingproduced on the eye.

The first sensor and the optional third sensor may have differentcharacteristics of speeds and resolution from the optional second andfourth sensors. In particular, the first sensor and the optional thirdsensor may operate at a frequency of between 10 Hz and 150 Hz, inparticular of about 30 images/sec and have a resolution of at least100*100 pixels, for example of between 300*300 pixels and 1280*960pixels.

The optional second sensor and the optional fourth sensor may operate ata frequency of greater than 200 Hz, in particular of between 200 Hz and2000 Hz, or indeed between 300 Hz and 2000 Hz, in particular equal to500 Hz and have a resolution of between 32*32 pixels and 140*140 pixels.

In order to minimize the accumulation of errors of the dynamicmeasurement and of its drift, the frequency ratio between the twosensors is preferably about 10 or more, that is to say the acquisitionfrequency of the second sensor is preferably equal to more than 10 timesthat of the first sensor.

Position of the User's Head

It is possible to determine the movement of the device with respect to abase module. The base module may for example be fixed to a screen of acomputing system.

The expression “computing system” is understood to mean any computingsystem comprising a screen for interacting with the user, for example afixed or portable computer, a tablet, a fixed or mobile telephone, thislist not being limiting.

It is for example possible to use optical beacons or luminous sourcesdisposed on the device, in particular fixed to the device, for exampleon branches of the latter, and a sensor, for example a camera, disposedon the base module.

Here the term “optical beacon” designates a passive element such as anoptical reflector or a material reflecting a certain wavelength when itis illuminated by an external source. Such beacons offer an alternativepossibility to the presence of active elements such as luminous sourcesof a certain wavelength belonging to the device.

The wavelength emitted directly by a luminous source of the device orreflected by a beacon may be in the visible, or the near infrared, whichis preferable since it is invisible to the human eye. This luminous fluxemitted directly by the luminous source of the device or reflected by abeacon may have one or more wavelengths simultaneously or alternatelyover time, may be polarized linearly, circularly, or in any other way,may be continuous in amplitude or in phase, or amplitude-modulated orphase-modulated.

The movement of the device with respect to the base module is determinedfor example by triangulation. It is for example possible to use at leastthree points defined by luminous beacons or sources [see your paragraphfurther on regarding the possible forms] on the device and a camera onthe base module.

The at least three points are not aligned and are disposed in a planenot perpendicular to the optical axis of said camera. The at least threepoints may be disposed in a plane substantially parallel to the opticalaxis of said camera. A corresponding algorithm is described in thearticle “3D Pose from 3 Corresponding Points under Weak-PerspectiveProjection” by T. D. Alter, Massachusetts Institute of Technology,Artificial Intelligence Laboratory, A. I. Memo No. 1378, July 1992.

The optical beacons or luminous sources may be of various forms.

They may be pointlike, for example of LED type, possibly between 3 and 9in number, each point then comprising an optical beacon or a luminoussource.

They may be of linear, curved, rectilinear, continuous or non-continuousform, for example comprise an optical fiber with lateral lighting.

The optical beacons or luminous sources may form an identifiable andnon-ambiguous pattern.

They may exhibit combinations of several forms.

The sources or beacons are for example disposed toward the outside ofthe head.

The luminous sources used to determine the movement of the device withrespect to the base module may be for example infrared LEDs. The cameramay be an infrared camera.

A reconstruction is undertaken of the six degrees of freedom of theposition of the head with respect to the base module, stated otherwisewith respect to the screen to which it may be fixed.

Thus, one measures on the one hand the direction of the gaze, statedotherwise the movement of the eyes with respect to the head, and on theother hand the movement of the head with respect to the screen. Such amethod makes it possible to minimize the necessary calibration. The twoprocessings may be performed simultaneously.

In a variant embodiment, it is for example possible to use twogyroscopes and a distance sensor, one disposed on the device and theother disposed on the base module, which may be fixed to a screen.

In a variant embodiment, the screen is not independent of the device butsecured to the latter, so that the determination of the direction of theuser's gaze with respect to the device suffices to determine thedirection of the user's gaze with respect to the screen. In anothervariant embodiment, the screen is remote from the device.

Observation Point

It is possible to determine the point of observation of the user on ascreen on which the base module is fixed, in particular a screen of thecomputing system mentioned above, on the basis of the meeting point ofthe two visual axes of the user determined on the basis of the items ofinformation relating to the direction of gaze. The expression “visualaxis” is understood to mean the axis between the fovea and the opticalcenter of the user's eye. The fovea is a zone of the retina slightlyoffset with respect to the center of the retina, and allowing the bestcolor vision. The meeting point of the two visual axes thereforeindicates the observation point on the screen. The “optical axis” is theaxis between the center of the user's retina and the optical center ofthe eye. A slight offset of about 5°, and which depends on the user,exists between these two axes. By using a precise measurement of therelative orientation of the head with respect to the screen, and ameasurement of the orientation of the two eyes, stated otherwise abinocular measurement, the invention makes it possible to obtain themeasurement of the convergent visual axes on the screen. The measurementof the direction of the optical axes does not call upon thereconstruction of the point of corneal reflection (glint) introducing anon-linearity having to be corrected by an unwieldy calibration. Thus,the invention makes it possible to facilitate the determination of theobservation point.

Calibration

It is possible to perform a calibration by asking the user to follow amobile target on the screen, stated otherwise the eye is compelled toperform an ocular tracking. In case of ocular tracking, the eyephysiologically optimizes its displacement so as to minimize it, anduses the same zone of the fovea, so that this calibration allows theposition of the fovea to be defined better. A variant consists in askingthe user to fix one or more points on the screen at precise instants. Inthe foregoing, the user is forewarned that a calibration is performed.

In the invention, no points of reflections of a luminous source, forexample of the sensors, are sought. It is not sought to chart a point ofreflection (glint) of a luminous source on the cornea. In the inventionneither is it sought to recognize in the captured images the image of areal object reflected on the user's cornea.

Device

The subject of the invention is furthermore, according to another of itsaspects, independently or in combination with the foregoing, a devicefor determining the direction of the gaze of a user, in particularspectacles or helmet, intended to be immobilized on the user's head, soas to control a computing system, comprising:

-   -   at least one first optical sensor configured to acquire an image        of all or part of an eye of the user,    -   an electronic circuit allowing at least one preprocessing of the        data, and    -   a wireless emitter.

The wireless transmission of the data makes it necessary to reduce thevolume of the data to be transmitted.

The wireless emitter may have receiving functions and be anemitter/receiver. The data transmitted may be analog or digital. Theseare for example data of audio or video type.

The processor makes it possible to accelerate the final processing ofthe data by virtue of the preprocessing. The use of an onboard processormakes it possible to perform a preprocessing of the data obtained fromthe sensors and therefore to decrease the quantity of data to betransmitted to the base module, thereby making it possible to acceleratethe method. The processor may comprise at least a part of the previouslymentioned algorithms that is necessary for the processing of the data.

The device is preferably positioned on the user's nose and/or ears, in amanner analogous to a pair of spectacles.

The device may comprise a battery affording it enough energyself-sufficiency, for example of at least several hours of operationwithout recharging, or indeed of at least a day. The use oflow-resolution images makes it possible to reduce electrical consumptionand thus to increase the energy self-sufficiency of the device.

The user's head may be stationary with respect to the device.

Assembly

The subject of the invention is furthermore, according to another of itsaspects, independently or in combination with the foregoing, an assemblycomprising:

-   -   a device such as described above, and    -   a base module intended to be fixed to a screen of a computing        system and connected to the latter.

The device may communicate with the base module through a wireless link.

The base module may be configured to be able to be used immediately uponits connection to the computing system. Stated otherwise, it may berecognized rapidly, easily and automatically by the operating system ofthe computing system immediately upon connection or immediately uponrebooting after hardware installation (“Plug and Play”). This procedureallows installation while requiring a minimum of intervention on thepart of the user and therefore while minimizing manipulation errors andparametrization errors.

The assembly may comprise several devices able to communicate with oneand the same base module.

DESCRIPTION OF THE FIGURES

The invention will be better understood on reading the detaileddescription which will follow, of an exemplary embodiment of theinvention, and on examining the appended drawing in which:

FIG. 1 is a schematic and partial perspective view of a device fordetermining the direction of gaze in accordance with the invention,

FIG. 2 illustrates in a schematic and partial manner an assemblycomprising the device of FIG. 1,

FIG. 3 is a block diagram illustrating the method for determining thedirection of gaze in accordance with the invention,

FIG. 4 illustrates the capture of the images on the eye, and

FIG. 5 illustrates in a schematic manner an assembly according to theinvention in a given environment.

Illustrated in FIG. 1 is a user U wearing a device 10 for determiningthe direction of gaze, taking the form of spectacles worn by the user,comprising branches 11 resting on the ears and a central part 12 restingon the nose, the lenses 13 of the spectacles being able to comprise ananti-reflection coating.

The device comprises in the example described two infrared LEDs 16disposed in the central part 12 on either side of the nose and eachoriented toward one of the user's eyes, as well as four RGB cameras 15a, 15 b that are able to detect infrared radiation. Each of the cameras15 a, 15 b is disposed and oriented toward one of the user's eyes, beingdisposed below the eyes on the perimeter of the lenses 13 of the deviceand each disposed otherwise than in a vertical plane passing through thecenter of the user's pupil. For each of the user's eyes, two cameras 15a, 15 b are disposed on either side of them, one on the side of the noseand the other on the side of the branch resting on the correspondingear, being oriented toward the user's corresponding eye so as to acquireimages thereof. The cameras 15 a are oriented toward the user's pupil,and the cameras 15 b toward the user's sclera.

The device 10 also comprises an electronic circuit 17 housed in theexample described in one of the branches 11 of the device, thiselectronic circuit 17 making it possible to process the images acquiredby the cameras 15 a, 15 b, as will be described further on.

The device furthermore comprises a battery, not visible in the figure,disposed for example in the second branch of the device and affording itenough energy self-sufficiency as not to have to be recharged for anacceptable duration, for example of several hours, or indeed of anentire day.

The device furthermore comprises a wireless emitter also housed in oneof the branches, transmitting the data to a base module 20 fixed to ascreen 25 of a computing system and connected to the latter, asillustrated in FIG. 2.

The device furthermore comprises luminous sources 19 which make itpossible, by virtue of a camera 22 disposed on the base module 20, todetermine the movement of the device 10 with respect to the base module20, in such a way as to determine a relative displacement of the eyewith respect to the camera and the evolution of the direction of gazeover time.

Said luminous sources 19 are, in the example described, infrared LEDS,for example disposed in a non-aligned manner and in a plane which is notperpendicular to an axis of the camera of the base module, asillustrated. In the example, one of the luminous sources 19 is disposedabove the user's nose, while the others are disposed on either side ofthe eyes in the upper part of the device. The luminous sources 19 areoriented toward the outside, that is to say toward a camera 22 of thebase module 20.

In a variant, optical beacons are used in place of the luminous sources19.

The steps, as illustrated in FIG. 3, of a method in accordance with theinvention will now be described in detail.

In a step 30, each of the cameras 15 a, 15 b captures images I atregular time intervals of the user's corresponding eye or more preciselyof the part of the corresponding user's eye.

In the example described, the cameras 15 a capture an image A of a zoneof the eye comprising at least partially the pupil while the sensors 15b capture an at least partial image B of the sclera of the eye, as maybe seen in FIG. 4. Of course, each of the sensors could capture imagesat one and the same time of the pupil and of the sclera without therebydeparting from the scope of the present invention.

The images I are processed in a step 40 at least partially to perform atleast in part two distinct processings (a) and (b), by means of theelectronic circuit 17 of the device 10, which makes it possible toperform these two processings at least in part, the processings beingthereafter continued in the base module. In the example described, onlya part of the two processings may be performed in the onboard processorembedded in the device, the data being thereafter transmitted by awireless link F to the base module in which the remainder of theprocessings is performed.

The first processing (a) makes it possible to provide an item ofinformation about the orientation of the eye on the basis of theobservation of a zone of the eye whose aspect varies with the rotationof the eye. This processing may be performed for each of the imagescaptured. It comprises the determination in step 41 of the parametersrelated to the shape of the ellipse corresponding to the pupil. Thefirst processing (a) makes it possible to deduce therefrom the opticalaxes in the frame of reference of the user's head in step 42.

More precisely, steps 41 and 42 of reconstructing the orientation of thepupil comprise the selecting of the darkest zone of the processed image,the isolating of the pupil in this image, and then the obtaining of thecontour of the pupil. From this are deduced the position of the centerof the pupil, the major and minor axes of the ellipse and an angledefining the orientation of the ellipse, and finally the normal to thepupil in a three-dimensional space.

Thereafter, a reconstruction of the center of rotation of the pupil isundertaken by determining a sphere on the basis of measurement pointsand of the previously determined normals. From this are deduced thecenter of rotation and the radius of this sphere.

Moreover, a second processing (b) of the images is performed whichprovides an item of information about the kinematics of the eye bycomparing at least two successive images in step 45.

For this purpose, a contrasted sub-image in the image I at a giveninstant is firstly selected. Next, a displacement of this same sub-imageis sought by seeking a corresponding sub-image in the following imagesuch that a correlation between the two sub-images is maximal. Theapproximation of the non-deformation of the sub-image between the twosuccessive images is correct insofar as the frequency of capture of theimages is sufficient, being in particular much greater than the speed ofthe observed movement, that is to say in the example described greaterthan 500 Hz.

Starting from a primary orientation of the eye, the change oforientation of the eye is calculated on the basis of the set ofpreviously determined infinitesimal displacements. The angulardisplacement of the eye, that is to say the two angles of rotation(pitch and yaw) around arbitrary perpendicular axes is thenreconstructed, from an arbitrary direction.

Moreover, the second processing (b) is used in step 47 to correct thecorneal refraction which may produce non-linearities stemming from therefraction of the pupil on the cornea, which make it possible to correctthe items of information of the first processing (a) in step 42. Thedirection of the normal obtained previously makes it possible todetermine a direction of the optical axis which may be marred by errordue to corneal diffraction, and noise stemming from the reconstructionof the ellipse. The angular displacement makes it possible to obtain thevariation of the angular variation of the optical axis on the basis ofan arbitrary initial direction. This measurement may also be marred byan error related to the resolution of the correlation measurement. Thedirection of the optical axis is obtained by weighting these twomeasurements at each iteration, by taking into account the positioningobtained at the previous step, the three-dimensional reconstruction ofthe eye, and a parametrization arising from the calibration.

In parallel, a processing 50 is performed to determine the position ofthe user's head with respect to the base module, that is to say themovement of the device 10 with respect to the base module 20. The camera22 disposed on the base module 20 captures images I of the user's head,in which images the light arising from at least three points defined bythe optical beacons or the luminous sources 19 of the device 10 isvisible. These images are processed so as to determine the position ofthe device with respect to the base module. In a first step 51, theluminous points in the image are selected. In a second step 52, thenoise is eliminated so as to ensure that the luminous points viewed inthe images do indeed correspond to the luminous beacons or sources 19intended to determine the movement of the device with respect to thebase module. In a third step 53, the position of the luminous points inthe image is determined and on the basis of these data, in a step 55,the position of the head with respect to the base module is calculated.

Moreover, in a step 60, the four angles of the fovea with respect to thecameras are determined on the basis of physiological data 61, of presetparameters 62, as well as of previously calibrated coordinates of thescreen 63. The preset parameters 62 may be construction parameters ofthe device and of the base module, for example the position of thecameras or of the lighting.

With calibration data 65, the two visual axes are deduced therefrom in astep 70.

Finally, on the basis of the crossover of the visual axes as well as ofthe position of the head with respect to the base module, thecoordinates of the point of the screen at which the user is gazing arecalculated in step 80, and this may be transmitted to the computingsystem in step 81.

EXAMPLES OF USE OF A DEVICE ACCORDING TO THE INVENTION Example 1:Evolution in a Given Environment

The device makes it possible to establish a relationship between thedirection of the gaze of an operator and a work environment so as toimprove the design and the ergonomics of this environment. Theenvironment being able to be for example a piloting cockpit of anaircraft, such as illustrated in FIG. 5, of a car, of a simulator or amulti-screen control environment.

The assembly 100 of FIG. 5 comprises a device 10 in the form ofspectacles, not illustrated, communicating through a wireless link withthe base module 20, the latter connected to a computing system 90. Theassembly 100 is disposed in an environment 200, here an aircraftpiloting cockpit. Thus the base module 20 may be used immediately uponits connection to the computing system. The assembly 100 representedcomprises several devices 10 and 10′, worn for example by the pilot andthe co-pilot. In addition to four image sensors and an electricalcircuit, none of which are represented, each device 10, 10′ comprises anemitter/receiver 40 for communicating wirelessly with the base 20, aninformation representation system 60 comprising in the exampleillustrated a semi-transparent screen 62 partially overlaid on thelenses of the spectacles, and an earpiece 63. The device 1 alsocomprises a sensor of an item of physiological information 70 so as toevaluate the psychological state of the wearer of the device 10, inparticular in a potentially stress generating emergency situation.

Example 2: Training and/or Gaming

In the case of “serious games” or operator training procedures, thedevice makes it possible to quantify the gaze and the effectiveness ofthe training and in particular to measure operator compliance with thesafety standards in a critical environment.

Example 3: User Interface

In the case of the handicapped, the device according to the inventionmay in particular act as substitute for the mouse and for the keyboardfor upper limb paraplegics.

With the binocular measurement (simultaneous measurement of thedirection of both eyes in the same frame of reference), the user'svergence is measured, the latter being a fundamental parameter havingapplications in the field of ophthalmology or three-dimensional spacecharting.

The implementations represented and the examples are given merely by wayof illustration and are not limiting of the invention.

The invention claimed is:
 1. A method for determining the direction ofthe gaze of a user comprising acquisition of images of the eye, with theaid of at least one optical sensor, the method comprising: a) a firstprocessing of acquired images which provides an item of informationabout the orientation of the eye, on the basis of the observation of azone of the eye whose aspect varies with the rotation of the eye, and b)a second processing of these acquired images independent of the firstprocessing which provides an item of information about the kinematics ofthe eye by comparing at least two successive images, the secondprocessing comprising determination of an optical flow between twosuccessive images of the eye, in which method an item of informationrelating to the direction of the gaze in relation to the user's head isgenerated on the basis at least of the items of information delivered bythe first and second processings, in which method use is made of adevice worn by the user comprising: at least one first optical sensorbeing a camera configured to acquire an image of all or part of an eyeof the user, an electrical circuit allowing at least one preprocessingof the acquired images, so as to determine a relative displacement ofthe eye with respect to the camera and the evolution of the direction ofgaze over time, in which the device comprises at least one secondoptical sensor configured to acquire an image of all or part of theuser's eye, in which the first and second optical sensors allow theacquisition of images of one and the same eye of the user.
 2. The methodas claimed in claim 1, in which the first processing comprises thedetermination of the parameters related to the shape of the ellipsecorresponding to pupil so as to determine the position of a point of theeye.
 3. The method as claimed in claim 1, the image acquisition schemeused being non-intrusive of the eye and the images acquired representingexternal kinematic and/or anatomical data of the eye.
 4. The method asclaimed in claim 1, the item of information delivered by the firstprocessing making it possible to correct within the precision of themeasurement the drifts related to the second processing.
 5. The methodaccording to claim 1, in which the first and the second optical sensorare each directed toward a different part of one and the same eye of theuser.
 6. The method as claimed in claim 1, furthermore comprising atleast one other optical sensor worn by the user and configured toacquire an image of all or part of the user's second eye.
 7. The methodas claimed in claim 6, in which the movement of the device is determinedwith respect to a base module.
 8. The method as claimed in claim 7, inwhich the point of observation of the user is determined on a screen towhich the base module is fixed, on the basis of the meeting point of twovisual axes of the user determined on the basis of the items ofinformation relating to the direction of the gaze.
 9. The method asclaimed in claim 1, in which a calibration is performed by asking theuser to follow a mobile target on a screen.
 10. A method for determiningthe direction of the gaze of a user comprising: acquisition of images ofan eye with a device worn by the user comprising: at least one firstoptical sensor being a camera configured to acquire an image of all orpart of the eye of the user, an electrical circuit allowing at least onepreprocessing of the acquired images, so as to determine a relativedisplacement of the eye with respect to the camera and the evolution ofthe direction of gaze over time, at least one second optical sensorconfigured to acquire an image of all or part of the user's eye, themethod comprising: a) a first processing of acquired images whichprovides an item of information about the orientation of the eye, on thebasis of the observation of a zone of the eye whose aspect varies withthe rotation of the eye, and b) a second processing of acquired imageswhich provides an item of information about the kinematics of the eye bycomparing at least two successive images, in which method an item ofinformation relating to the direction of the gaze in relation to theuser's head is generated on the basis at least of the items ofinformation delivered by the first and second processings and in whichthe first and second optical sensors allow the acquisition of images ofone and the same eye of the user.